• Like a “scatter” plot
• Hexagons are equally distributed in all directions

Too cool when it’s warm

Too warm when it’s cooler

What is skill (RMSE)?

\(\sqrt{\frac{1}{N}\sum\limits_{1}^{N} \left(\overline{f} - o\right)^{2}}\)

What is spread?

\(\sqrt{\frac{1}{N}\sum\limits_{1}^{N} \left(Var\left(f\right)\right)}\)

What is skill (STDE)?

\(\sqrt{\frac{1}{NM-1}\sum\limits_{i=1}^{NM} \left(f_{i} - o\right)^{2}}\)

What is spread?

\(\sqrt{\frac{1}{N}\sum\limits_{1}^{N} \left(Var\left(f\right)\right)}\)

• Shows dispersion of ensemble
• Aiming for normalized frequency of 1 for each rank

• Area between forecast CDF and observed step function
• Averaged over each case
• Equivalent to mean absolute error for deterministic

• \(BS=\frac{1}{N}\sum\limits_{i=1}^{N}\left(f_i-o_i\right)^2\)
• Range [0, \(\infty\)]
• Perfect score = 0
• Sensitive to event rarity

• Compares Brier Score with a reference (usually sample climatology)
• \(BSS=1-\frac{BS}{BS_{ref}}\)
• Range [\(-\infty\), 1]
• \(<=0\) -> No skill
• 1 -> Perfect skill
• Reduces to \(-\infty\) when there are no events

• With what frequency did the event occur for a given forecast probability?
• Observed frequency should equal forecast probability
• Easy to interpret and communicate
• Difficult to reduce to a single number

• Given that the event occurred, how did the forecast do for each probability?
• \(HR=\frac{H}{H+M}\)
• \(FAR=\frac{FA}{FA + CR}\)
• Top left is best
• Can be reduced to area under the ROC curve

• \(BS_{rel}-BS_{res}+BS_{unc}\)
• Perfect \(BS_{rel} = 0\)
• Perfect \(BS_{res}=BS_{unc}\)
• \(BS_{unc}\) is a function of the observations only

• For what cost/loss ratios is the ensemble better than climatology
• Usually sample climatology